Haptically Enabled User Interface

ABSTRACT

A device has a user interface that generates a haptic effect in response to user inputs or gestures. In one embodiment, the device receives an indication that the user is scrolling through a list of elements and an indication that an element is selected. The device determines the scroll rate and generates a haptic effect that has a magnitude that is based on the scroll rate.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of co-pending U.S. Patent ApplicationNo. 12/046,934, filed Mar. 12, 2008, entitled “Haptically Enabled UserInterface,” the entirety of which is hereby incorporated by referenceherein.

FIELD OF THE INVENTION

One embodiment of the present invention is directed to haptic effects.More particularly, one embodiment of the present invention is directedto a user interface having haptic effects.

BACKGROUND INFORMATION

Electronic device manufacturers strive to produce a rich interface forusers. Conventional devices use visual and auditory cues to providefeedback to a user. In some interface devices, kinesthetic feedback(such as active and resistive force feedback) and/or tactile feedback(such as vibration, texture, and heat) is also provided to the user,more generally known collectively as “haptic feedback” or “hapticeffects”. Haptic feedback can provide cues that enhance and simplify theuser interface. Specifically, vibration effects, or vibrotactile hapticeffects, may be useful in providing cues to users of electronic devicesto alert the user to specific events, or provide realistic feedback tocreate greater sensory immersion within a simulated or virtualenvironment.

Haptic feedback has also been increasingly incorporated in portableelectronic devices, such as cellular telephones, personal digitalassistants (PDAs), portable gaming devices, and a variety of otherportable electronic devices. For example, some portable gamingapplications are capable of vibrating in a manner similar to controldevices (e.g., joysticks, etc.) used with larger-scale gaming systemsthat are configured to provide haptic feedback. Additionally, devicessuch as cellular telephones and PDAs are capable of providing variousalerts to users by way of vibrations. For example, a cellular telephonecan alert a user to an incoming telephone call by vibrating. Similarly,a PDA can alert a user to a scheduled calendar item or provide a userwith a reminder for a “to do” list item or calendar appointment.

Increasingly, portable devices are moving away from physical buttons infavor of touchscreen-only interfaces. This shift allows increasedflexibility, reduced parts count, and reduced dependence onfailure-prone mechanical buttons and is in line with emerging trends inproduct design. Many of these touchscreen devices include sophisticateduser interfaces that convert user gestures, including multi-touchgestures, into input commands.

SUMMARY OF THE INVENTION

One embodiment is a device having a user interface that generates ahaptic effect in response to user inputs or gestures. In one embodiment,the device receives an indication that the user is scrolling through alist of elements and an indication that an element is selected. Thedevice determines the scroll rate and generates a haptic effect that hasa magnitude that is based on the scroll rate.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a cellular telephone in accordance with oneembodiment.

FIG. 2 is a block diagram of a user interface that includes a list ofelements that can be scrolled through the interaction of a scroll bar inaccordance with one embodiment.

FIG. 3 is a flow diagram of the functionality performed by the telephoneof FIG. 1 in accordance with one embodiment in response to a scrollingof elements through a user interface.

FIG. 4 is a flow diagram of the functionality performed by the telephoneof FIG. 1 in accordance with one embodiment in response to a flickedobject.

FIG. 5 is a user interface for a touchscreen that provides a virtualslider for controlling volume.

FIG. 6 is a flow diagram of the functionality performed by the telephoneof FIG. 1 in accordance with one embodiment in response to the movementof a slider.

DETAILED DESCRIPTION

One embodiment is a haptically enabled user interface that provideshaptic confirmation to a user in response to various user inputs andgestures or device activity.

FIG. 1 is a block diagram of a cellular telephone 10 in accordance withone embodiment. Telephone 10 includes a touchscreen 11 and may includemechanical keys/buttons 13. Internal to telephone 10 is a hapticfeedback system that generates vibrations on telephone 10. In oneembodiment, the vibrations are generated on the entire telephone 10. Inother embodiments, specific portions of telephone 10 can be hapticallyenabled by the haptic feedback system, including the entire touchscreen11 or individual portions of touchscreen 11.

The haptic feedback system includes a processor 12. Coupled to processor12 is a memory 20 and an actuator drive circuit 16, which is coupled toa vibration actuator 18. Processor 12 may be any type of general purposeprocessor, or could be a processor specifically designed to providehaptic effects, such as an application-specific integrated circuit(“ASIC”). Processor 12 may be the same processor that operates theentire telephone 10, or may be a separate processor. Processor 12 candecide what haptic effects are to be played and the order in which theeffects are played based on high level parameters. In general, the highlevel parameters that define a particular haptic effect includemagnitude, frequency and duration. Low level parameters such asstreaming motor commands could also be used to determine a particularhaptic effect. A haptic effect may be considered “dynamic” if itincludes some variation of these parameters when the haptic effect isgenerated or a variation of these parameters based on a user'sinteraction.

Processor 12 outputs the control signals to drive circuit 16 whichincludes electronic components and circuitry used to supply actuator 18with the required electrical current and voltage to cause the desiredhaptic effects. Actuator 18 is a haptic device that generates avibration on telephone 10. Actuator 18 can include one or more forceapplying mechanisms which are capable of applying a vibrotactile forceto a user of telephone 10 (e.g., via the housing of telephone 10).Actuator 18 may be, for example, an electromagnetic actuator, anEccentric Rotating Mass (“ERM”) in which an eccentric mass is moved by amotor, a Linear Resonant Actuator (“LRA”) in which a mass attached to aspring is driven back and forth, or a “smart material” such aspiezoelectric, electro-active polymers or shape memory alloys. Memorydevice 20 can be any type of storage device or computer-readable medium,such as random access memory (“RAM”) or read-only memory (“ROM”). Memory20 stores instructions executed by processor 12. Memory 20 may also belocated internal to processor 12, or any combination of internal andexternal memory.

Touchscreen 11 recognizes touches, and may also recognize the positionand magnitude of touches on a touch sensitive surface. The datacorresponding to the touches is sent to processor 12, or anotherprocessor within telephone 10, and processor 12 interprets the touchesand in response generates haptic effects. Touchscreen 11 may sensetouches using any sensing technology, including capacitive sensing,resistive sensing, surface acoustic wave sensing, pressure sensing,optical sensing, etc. Touchscreen 11 may sense multi-touch contacts andmay be capable of distinguishing multiple touches that occur at the sametime. Touchscreen 11 may further display images for the user to interactwith, such as keys, dials, etc., or may be a touchpad with minimal or noimages.

Although the embodiment of FIG. 1 is a cellular telephone 10, otherembodiments may be any type of device that provides a user interface andis capable of generating haptic effects. The device may be handheld andmay include a touchscreen that generates the user interface. In otherembodiments, rather than generating haptic effects on a handheld devicefor touchscreen interactions, the device may be a computer system thatincludes a display and a cursor control device such as a mouse,touchpad, mini-joystick, etc. The display displays the user interfaceand the mouse or other device includes vibration actuator 18 so that thehaptic effect is generated on the device as it is held by the user.

FIG. 2 is a block diagram of a user interface (“UI”) 200 that includes alist of elements 202 that can be scrolled through the interaction of ascroll bar 204 in accordance with one embodiment. A user can slidescroll bar 204 through interaction with a touchscreen, or by pressing abutton or using a mouse or other interface device. Further, arrows 206can be selected in a known manner to scroll the list of elements 202.

When elements 202 are being scrolled, one of the elements is highlightedto indicate which element is “selected”. In one embodiment, a hapticeffect is generated when each element of elements 202 is selected. Thehaptic effect may be in the form of a “bump” “pop”, “click”, “tick”,etc. through a predetermined combination of magnitude, frequency andduration. However, as the scroll rate is increased, a constant magnitudehaptic effect can feel like a constant “buzz” to the user and fail toprovide meaningful information. Therefore, in one embodiment, the hapticeffect volume/magnitude is decreased as the scroll rate increases, andvice versa. This keeps the overall “haptic energy” of the UI interactionat a low and unobtrusive level.

FIG. 3 is a flow diagram of the functionality performed by telephone 10of FIG. 1 in accordance with one embodiment in response to a scrollingof elements 202 through a user interface. In one embodiment, thefunctionality of FIG. 3, and FIGS. 4 and 6 below, is implemented bysoftware stored in memory and executed by a processor. In otherembodiments, the functionality can be performed by hardware, or anycombination of hardware and software.

At 302, an indication that the user is scrolling through the list ofelements 202 and one of the elements has been highlighted or selected isreceived. In one embodiment, the list of elements 202 can be a literallist such as a menu list or list of contacts as in FIG. 2, or it couldbe an abstract list such as a 30 cylinder clock with a list of scrollingdigital numbers that can be changed by scroll/flick tumbling of numbers.The scrolling may be accomplished by interacting with the screen througha touchscreen, by button press/hold events in up/down, left/right,diagonal or circular motion, or by any other method.

At 304, the current scroll rate of elements 202 is determined and may becompared to a previous scroll rate. It is determined if the currentscroll rate is increasing or decreasing.

At 306, a haptic effect that corresponds to the selection of one of theelements is generated based on the current scroll rate and whether it isincreasing or decreasing or based on the magnitude of the scroll rate.In one embodiment, the haptic effect has a short duration and repeatswhenever a new element is selected. The magnitude of the haptic effectis decreased relative to a previous haptic effect if the scroll rate isincreasing. Similarly, the magnitude of the haptic effect is increasedrelative to a previous haptic effect if the scroll rate is decreasing.In another embodiment, the magnitude of the haptic effect is determinedfrom a lookup table that bases the magnitude inversely on the currentscroll rate. The greater the scroll rate, the smaller the magnitude, andvice versa.

In one embodiment, when a user is scrolling the list of elements 202,eventually the user will reach the end of the list. At the point, thescrolling may stop or the list may wrap around to the beginning of thelist. In one embodiment, a haptic effect will be generated when the lastitem in the list is selected or when the list wraps around so that auser receives a non-visual indication. This haptic effect is differentthan the haptic effect that is generated when one of the items isselected that is not at the end of the list, as disclosed above. In oneembodiment, the end-of-list haptic effect differs via a change in anycombination of duration, amplitude, frequency, etc.

In another embodiment, a list of items or other objects, such a photos,a ball, puck, etc. can be “flicked” using a gesture so that the listvisually moves. The speed of the movement can be dependent on the speedof the flick. Upon hitting a virtual stop, like an end-of-list, wall orother “hard object,” the flicked object reacts visually with a bounce.In one embodiment, the bounce causes a corresponding haptic effect andoptionally an audible output. In one embodiment, the haptic effect wouldhave characteristic parameters of being initially very strong (i.e.,high magnitude) with a rapidly decreasing magnitude as the bouncedobject comes to rest. Further, another haptic effect may be generated,such as a subtle haptic click, pop, or tick effect to confirm that theflicked object has stopped moving and come to rest.

FIG. 4 is a flow diagram of the functionality performed by telephone 10of FIG. 1 in accordance with one embodiment in response to a flickedobject. At 402, processor 12 receives an indication that a flickedobject has reached an end point, such as a wall, end-of-list, etc. At404, in response to the end point indication, a haptic effect isgenerated. In one embodiment, the haptic effect is dynamic in that itinitially has a high magnitude, and then has a rapidly decreasingmagnitude as the flicked object comes to rest.

In another embodiment, a user interface simulates a slider. There are anumber of physical control sliders found in professional audio andindustrial control equipment. These sliders are generally controlled bydragging the slider with a fingertip. Many of the slider controlfunctionality can be built into touchscreen UIs. FIG. 5 is a UI 500 fora touchscreen that provides a virtual slider 502 for controlling volume.UI 500 provides information about the position or state of virtualslider 502 and the parameter it is controlling. Slider 502 may bemanipulated by a finger or a cursor. Other UIs may provide a slider thatis laid out in a up/down orientation or might be rotational such as aknob or wheel,

UI 500 has a minimum and maximum setting, and in one embodiment adynamic haptic effect is generated that has an increasing magnitude asthe volume is increased and a decreasing magnitude as the volume isdecreased. This type of haptic effect helps communicate the relativevolume of the parameter being increased/decreased, whether it is anaudible volume from a stereo system or a physical volume such as for anindustrial flow control system managing liquid volumes. In addition,positional haptic effects may be generated that simulate bumps or dentsthat might be used as slider end-stops, a centering location, or otherimportant positional locations in the slider that might be specific tothe end-user application being controlled (e.g., positions 504 and 506).

FIG. 6 is a flow diagram of the functionality performed by telephone 10of FIG. 1 in accordance with one embodiment in response to the movementof a slider. At 602, processor 12 receives an indication the slider hasmoved and an indication as to whether the parameter (e.g., volume) isdecreasing or increasing. At 604, in response to the parameterincreasing or decreasing, a dynamic haptic effect is generated that hasan increasing or decreasing magnitude or other parameter.

In another embodiment, slider 502 is a toggle type slider that has onlytwo resting positions or a limited number of positions, such as anon/off switch. In this embodiment, the haptic effect is generated toconfirm the action of the toggled control. Further, two or moredistinctly different haptic effects can communicate to the user intowhich state the slider has been placed, such as on/off,slow/medium/fast, etc. The differences in the haptic effect may be avaried magnitude, frequency, duration or any combination of these threeparameters.

In another embodiment, telephone 10 generates haptic effects duringscreen transitions. The haptic effects in one embodiment aresynchronized with any visual transitions occurring on screen. Examplesof screen transitions that may generate haptic effects include changesin screen orientation where the on-screen content rotates, content beingreplace with new content through screen fades, wipes, dissolves or othervideo transition techniques, changes in viewing size such as zooming inor out, panning of content such as web pages, pictures or documents,etc. In one embodiment, the haptic effect will be generated during thescreen transition, and a different haptic effect can be generated whenthe screen transition is completed.

In another embodiment, haptic effects are generated when a user dragsselected text or other items across a touchscreen. A haptic bump, pop ortick may be generated during the drag. This haptic effect could beplayed over each letter selected and a different, possibly stronger,effect could be played as whole words are selected in order to confirmboth individual letter and whole word selections. Further, a hapticeffect may be generated as each individual object becomes selectedduring the dragging. Further, while dragging an object or otherwisemoving a finger across a touchscreen, a haptic effect can be generatedto simulate the “surface” of the screen. For example, if the screen isdisplaying a bumpy road, the haptic effects can be generated so that itfeels as if the finger is dragging across a bumpy road.

In another embodiment, a haptic effect such as a bump, pop or tick couldbe generated as a user click-drags his finger or cursor across multipleobjects like words, files, directories, pictures, icons, etc. In oneembodiment, the haptic effect would be generated as each individualobject became selected or deselected from the grouping while dragging.This selection method could be accomplished with both single andmulti-touch touchscreens.

In another embodiment, a haptic effect such as a bump, pop or tick couldbe added as a user double-taps his finger or double-clicks a cursor toselect words. This confirmation method could be used when triple-tappingor triple-clicking to select whole sentences or paragraphs.

In some embodiments, a user is forced to wait while a device such astelephone 10 initializes, downloads content, etc. During this time theUI generally cannot be interacted with and any time spent attempting tointeract is wasted. In one embodiment, a haptic effect is generated toinform the user that the device has entered or has exited aloading/standby state that causes the user to wait.

In one embodiment, the haptic effect while the UI is unresponsive is asubtle, constant effect having a non-changing magnitude or frequencythat simply ends when the UI becomes available again for interactions.In another embodiment, the haptic effect is a dynamic effect thatincreases in magnitude and/or frequency as the system progresses towardscompletion of its task. Further, in one embodiment, a separateconfirming completion haptic effect can be generated, such as a bump,pop or tick when the UI becomes available again.

In another embodiment, a gesture from a user on touchscreen 11 can beused to unlock content that has previously been locked by device 10.Device 10 can be unlocked when a predetermined gesture is input by theuser. For example, the gesture can function as a password or pass codeto unlock the menu system of device 10. Examples of a predeterminedgesture include a swirl, swipe, press/tap pattern, or any combination ofthese gestures.

In one embodiment, a haptic effect is generated that corresponds to oris a representation of the gesture so the user may confirm the gestureor learn the gesture. The haptic effect can be dynamic and may beapplied to individual portions of touchscreen 11 at a time in order tosimulate directional motion such as a circular motion. This especiallyhas value when the gesture is a substitute for a pass code and the userneeds to memorize the gesture in order to access device 10. Without thehaptic effect, gestures such as a swipe, swirl or tap pattern may bereviewed/confirmed by replaying the gesture with visual and/or audiorepresentations of the gesture. However, the haptic effect can aid theuser in remembering the gesture pass code better than with only visualand/or audible cues, especially with a gesture that may include fingerpressure data patterns generated by touchscreen 11. The haptic effectcan “simulate” the finger pressure data by changing the amplitude valuesof the haptic effect. The haptic effect may also be generated for a userwho may have forgotten their gesture pass code—the playing of the hapticeffect will serve as a reminder.

In another embodiment, the user gesture is used as a “hot key” or macroto bypass a number of steps to arrive at a desired menu or function. Forexample, swirling a user's finger clockwise on touchscreen 11 twicemight indicate that the user wants to call home, or type a SMS mobilemessage. Device 10 would then bypass the standard menu selections thatwould otherwise be needed to be navigated and simply initiate the taskimmediately. As above, a haptic effect is generated to confirm thegesture to the user or allow the user to review the gesture.

As disclosed, a device having a user interface generates various hapticeffects in order to provide feedback to a user input or provideinformation regarding a state of the device. The haptic effects enhancethe usability of the device.

Several embodiments are specifically illustrated and/or describedherein. However, it will be appreciated that modifications andvariations are covered by the above teachings and within the purview ofthe appended claims without departing from the spirit and intended scopeof the invention.

What is claimed is:
 1. A system comprising: a processor; and a memory onwhich instructions executable by the processor are stored to cause theprocessor to: receive an indication that an object has been flicked by auser gesture and that the object has reached an end point that causesthe object to bounce; cause a haptic effect to be output when the objecthas reached the end point, wherein the haptic effect decreases inmagnitude as the bounced object comes to rest; and cause a second hapticeffect to be output to confirm that the bounced object has come to rest.2. The system of claim 1, wherein the memory further comprisesinstructions executable by the processor to cause the processor todetermine a speed of the flick, wherein a speed of the object isdetermined based on the speed of the flick.
 3. A device comprising: aprocessor; and a memory on which instructions executable by theprocessor are stored to cause the processor to: store a gesture on thedevice, wherein the gesture comprises a user interaction associated witha directional motion on a touchscreen, and cause a dynamic haptic effectthat simulates the gesture to be output, wherein the dynamic hapticeffect is applied to different individual portions of the touchscreen inseries to simulate the directional motion.
 4. The device of claim 3,wherein causing the dynamic haptic effect to be output comprises causinga haptic effect that changes amplitude to simulate a varied pressureportion of the gesture to be output, wherein the varied pressure portionincludes a varied finger pressure data pattern associated with thegesture.
 5. The device of claim 3, wherein the memory further comprisesinstructions executable by the processor to cause the processor tounlock the device based on the gesture.
 6. The device of claim 3,wherein the memory further comprises instructions executable by theprocessor to cause the processor to initiate a task based on thegesture.
 7. The device of claim 3, wherein the gesture comprises atleast one of a swirl pattern, a swipe pattern, or a tap pattern.
 8. Thedevice of claim 3, wherein the directional motion of the gesturecomprises a circular motion.
 9. The device of claim 3, wherein thememory further comprises instructions executable by the processor tocause the processor to cause the dynamic haptic effect to be output by:causing a first haptic effect to be output at a first portion of thetouchscreen at a first time; and causing a second haptic effect to beoutput at a second portion of the touchscreen at a second time, whereinthe second haptic effect is different from the first haptic effect, thefirst portion is different from the second portion, and the first timeis different from the second time.
 10. The device of claim 3, whereinthe memory further comprises instructions executable by the processor tocause the processor to: receive an input on the touchscreen, compare theinput on the touchscreen to the stored gesture; and cause the dynamichaptic effect to be output to confirm that the input matches the storedgesture.
 11. A non-transitory computer-readable medium comprisingprogram code, which when executed by a processor is configured to causethe processor to: receive an indication that an object has been flickedby a user gesture and that the object has reached an end point thatcauses the object to bounce; cause a haptic effect to be output when theobject has reached the end point, wherein the haptic effect decreases inmagnitude as the bounced object comes to rest; and cause a second hapticeffect to be output to confirm that the bounced object has come to rest.12. The non-transitory computer-readable medium of claim 11, furthercomprising instructions executable by the processor to cause theprocessor to determine a speed of the flick, wherein a speed of theobject is determined based on the speed of the flick.
 13. Anon-transitory computer-readable medium comprising program code, whichwhen executed by a processor of a device is configured to cause theprocessor to: store a gesture on the device, wherein the gesturecomprises a user interaction associated with a directional motion on atouchscreen, and cause a dynamic haptic effect that simulates thegesture to be output, wherein the dynamic haptic effect is applied todifferent individual portions of the touchscreen in series to simulatethe directional motion.
 14. The non-transitory computer-readable mediumof claim 13, wherein causing the dynamic haptic effect to be outputcomprises causing a haptic effect that changes amplitude to simulate avaried pressure portion of the gesture to be output, wherein the variedpressure portion includes a varied finger pressure data patternassociated with the gesture.
 15. The non-transitory computer-readablemedium of claim 13, further comprising instructions executable by theprocessor to cause the processor to unlock the device based on thegesture.
 16. The non-transitory computer-readable medium of claim 13,further comprising instructions executable by the processor to cause theprocessor to initiate a task based on the gesture.
 17. Thenon-transitory computer-readable medium of claim 13, wherein the gesturecomprises at least one of a swirl pattern, a swipe pattern, or a tappattern.
 18. The non-transitory computer-readable medium of claim 13,wherein the directional motion of the gesture comprises a circularmotion.
 19. The non-transitory computer-readable medium of claim 13,further comprising instructions executable by the processor to cause theprocessor to cause the dynamic haptic effect to be output by: causing afirst haptic effect to be output at a first portion of the touchscreenat a first time; and causing a second haptic effect to be output at asecond portion of the touchscreen at a second time, wherein the secondhaptic effect is different from the first haptic effect, the firstportion is different from the second portion, and the first time isdifferent from the second time.
 20. The non-transitory computer-readablemedium of claim 13, further comprising instructions executable by theprocessor to cause the processor to: receive an input on thetouchscreen, compare the input on the touchscreen to the stored gesture;and cause the dynamic haptic effect to be output to confirm that theinput matches the stored gesture.